Method for the 3-Dimensional Measurement of a Sample With a Measuring System Comprising a Laser Scanning Microscope and Such Measuring System

ABSTRACT

The invention relates to a method for the 3-dimensional measurement of a sample with a measuring system having a 3-dimensional measuring space and comprising a laser scanning microscope, characterised by—providing the measuring system with a 3-dimensional virtual reality device,—creating the 3-dimensional virtual space of the measuring space using the 3-dimensional virtual reality device,—allowing for selecting an operation in the virtual space,—providing real-time unidirectional or bidirectional convection between the measuring space and the virtual space such that an operation selected in the virtual space is performed in the measuring space and data measured in the measuring space is displayed in the virtual space. The invention further relates to a measuring system for the 3-dimensional measurement of a sample, the measuring system having a 3-dimensional measuring space and comprising a laser scanning microscope, characterised by further comprising a 3-dimensional virtual reality device for displaying a 3-dimensional virtual space of the measuring space, and a real-time unidirectional or bidirectional connection is provided between the laser scanning microscope and the 3-dimensional virtual reality device.

The present invention relates to a method for the 3-dimensionalmeasurement of a sample with a measuring system having a 3-dimensionalmeasuring space and comprising a laser scanning microscope. Theinvention further relates to such a measuring system.

According to the state of the art many 3-dimensional imaging devicesexist (CT, MRI, ultrasound, various laser scanning methods, e.g.confocal microscope, 2-photon microscope, 3-dimensional 2-photonmicroscope, spinning disc confocal microscope, atomic force microscope).These devices are generally connected to conventionalcomputer-configurations, i.e. the 3-dimensional (3D) informationresulting from the measurement is displayed as a 2-dimensional (2D)projection on a monitor. In medical, biological application fastperception of the information and fast decision making is importantbecause biological samples often have a short life-span or the measuredphenomenon can only be observed within a limited time period. In case ofsuch samples it is often an objective to change the examined sample byperforming surgical interaction, or an experiment. In some instancesstereoscopic displays are used in order to enhance viewing of theobtained data, however the viewing is separated in time from themeasurement and the physical interaction performed on the sample.

The 3-dimensional measurement of biological samples can be performed by3-dimensional (3D) laser scanning microscopes which carry out themeasurement by scanning the sample from point to point. The 3D laserscanning technologies are very important in analysing biologicalsamples, in particularly in imaging 3-dimensional biological structuresand tracing alteration of such structure on different time scales.

Commonly used 3D laser scanning microscopes are either confocalmicroscopes or two-photon microscopes. In the confocal microscopetechnology a pinhole is arranged before the detector to filter out lightreflected from any other plane than the focus plane of the microscopeobjective. Thereby it is possible to image planes lying in differentdepths within a sample (e.g. a biological specimen).

Two-photon laser scanning microscopes use a laser light of lower energyof which two photons are needed to excite a flourophore in a quantumevent, resulting in the emission of a fluorescence photon, which is thendetected by a detector. The probability of a near simultaneousabsorption of two photons is extremely low requiring a high flux ofexcitation photons, thus two-photon excitation practically only occursin the focal spot of the laser beam, i.e. a small ellipsoidal volumehaving typically a size of approximately 300 nm×300 nm×1000 nm.Generally a femtosecond pulsed laser is used to provide the requiredphoton flux for the two-photon excitation, while keeping the averagelaser beam intensity sufficiently low.

When applying either of the above-mentioned technologies the 3D scanningcan be carried out by moving the sample stage e.g. via stepping motors,however this is complicated to implement when using submerge specimenchambers or when electrical recording is performed on the biologicalspecimen with microelectrodes. Accordingly, in the case of analysingbiological specimens it is often preferred to move the focus spot of thelaser beam instead of moving the specimen. This can be achieved bydeflecting the laser beam to scan different points of a focal plane (XYplane) and by displacing the objective along its optical axis (Z axis)e.g. via a piezo-positioner to change the depth of the focal plane.Several known technologies exist for deflecting the laser beam prior toit entering the objective, e.g. via deflecting mirrors mounted ongalvanometric scanners, or via accousto-optical deflectors.

Further possibility is the use of so called holographic microscopy,where the desired 2- or 3-dimensional scanning effect is achieved byusing a spatial light modulator (SLM) (Volodymyr Nikolenko, Brendon O.Watson, Roberto Araya, Alan Woodruff, Darcy S. Peterka and Rafael Yuste,2008, Frontiers in Neuronal Circuits). A new technology for performing3D scanning which can be used in combination with the prior systems isthe so called spatiotemporal multiplexing microscopy, i.e. spatialimpulse separation (Adrian Cheng, J Tiago Gonçalves, Peyman Golshani,Katsushi Arisaka, Carlos Portera-Cailliau, 2011, Nature Methods). Inthis method a single laser impulse is divided by beam splitters intomore than one portions and each sub-impulse is focused to differentplanes with different time delay, because each one traverses differentimaging systems that image the sub-impulses into different focal planes.For the purpose of spatial scanning the sub-impulses that impactsequentially in time are separated by fast photon counting detectors.

The galvanometric scanners and the acousto-optical deflectors are veryfast devices, hence moving the focus spot to a desired XY plane positionand obtaining measurement data via the detector in that position can becarried out in less than 1 μs. However, due to the inertia of themicroscope objective the Z positioning takes substantially more time,rendering the 3D scanning a lengthy operation.

In order to achieve the signal per noise ratio commonly accepted in theart and supposing an average objective and an average sample size themeasurement can take many minutes (e.g. scanning a volume of 512×512×200pixels with a 2-photon microscope preferably with a resolution that isgreater than the optical resolution the measurement may take 5-20minutes). However, fast physiological reactions taking place inbiological samples are in the order of ms (e.g. action potential,synaptic signal transmission). The measuring time of scanningmicroscopes can be decreased by measuring only along the relevantregions, curves, points leaving out other parts of the sample. Such atechnology is disclosed for example in WO2010/007452. However theapplication of this and similar technologies requires first the spatialselection of a portion of the 3D sample.

A technology is known from the prior art (see Katona et al.: RollerCoaster Scaning reveals spontaneous triggering of dendiritc spikes inCA1 interneurons. PNAS, Feb. 1, 2011, vol. 108, no. 5) wherein thesample is scanned in planes orthogonal to the optical axis of the laserscanning microscope and the 2D sections are displayed one after theother to the user simulating forward or backward movement along theoptical axis. The users have to select the configuration (regions,curves, points) to be measured on the 2D sections.

The known method has a number of disadvantages: first of all it isdifficult to orientate oneself in a typical biologic object that has acomplicated spatial structure as in the case of various nerve-celltypes. The problem arises in particular when the objects areperpendicular to the scanned planes and when they are in the proximityof each other, in such cases it is difficult to trace the continuousobjects and to distinguish the separate objects between two neighbouringplanes (see e.g. electron microscope reconstruction programs). Thesituation is further aggravated if poor quality, low signal per noiseratio images need to be analysed or continuity and distinctness ofobjects have to be determined in such images.

A further disadvantage is that very often there is not enough time toview the 2D sections because of the locally permanently changingposition of the sample. For example if the user would like to select theideally required several hundreds of measuring points in 3D this processwould require such a long time that the sample would loose its originalposition even before finishing the selection to such an extent that thepoints selected at the beginning would no longer be in the correctposition.

It is an objective of the present invention to provide a method and ameasuring system for the analysis of 3-dimensional samples using a laserscanning microscope that overcomes the problems associated with theprior art.

It has been realised that if the measuring space of a laser scanningmicroscope (or other physical operation device) is displayed by a3-dimensional virtual reality device and a real-time connection isprovided between the measuring space and the 3-dimensional virtual spacethe above disadvantages can be overcome because 3-dimensional viewing ismore natural for a user conducting the measurement and even worsequality images with lower signal per noise ratio can be betterinterpreted and continuity and distinctness can be determined moreeasily.

Accordingly, the objective of the invention is achieved by a method forthe 3-dimensional measurement of a sample with a measuring system havinga 3-dimensional measuring space and comprising a laser scanningmicroscope, which method is characterised by

-   -   providing the measuring system with a 3-dimensional virtual        reality device,    -   creating the 3-dimensional virtual space of the measuring space        in a real space region that is spaced from the measuring space        using the 3-dimensional virtual reality device,    -   allowing for selecting an operation in the virtual space,    -   providing real-time bidirectional connection between the        measuring space and the virtual space such that an operation        selected in the virtual space is performed in the measuring        space and data measured in the measuring space is displayed in        the virtual space.

The above objective is further achieved by providing a measuring systemfor the 3-dimensional measurement of a sample, the measuring systemhaving a 3-dimensional measuring space and comprising a laser scanningmicroscope. The measuring system is characterised by comprising a3-dimensional virtual reality device for displaying a 3-dimensionalvirtual space of the measuring space in a real space region that isspaced apart from the measuring space, and a real-time bidirectionalconnection is provided between the laser scanning microscope and the3-dimensional virtual reality device.

Advantageous embodiments of the invention are defined in the attacheddependent claims.

Further details of the invention will be apparent from the accompanyingfigures and exemplary embodiments.

FIG. 1 is a schematic diagram of an exemplary measuring system accordingto the invention.

FIG. 2 a is a schematic diagram of another preferred embodiment of themeasuring system according to the invention.

FIG. 2 b is a schematic diagram of another preferred embodiment of themeasuring system according to the invention.

FIG. 2 c is a schematic diagram of a further preferred embodiment of themeasuring system according to the invention.

FIG. 1 is a schematic diagram of a measuring system 10 according to theinvention. The measuring system 10 has a 3-dimensional measuring space12 wherein a schematically illustrated sample 14 has been placed. Themeasuring space 12 is the spatial region within which physical operationcan be performed on the sample 14. Under physical operation themeasurement of physical parameters as well as physical interaction ismeant. Hence, the measuring space 12 is that spatial region of themeasuring system 10 within which a physical parameter of the sample 14can be measured or within which it is possible to physically interactwith the sample 14. In the context of the present invention physicalparameter is understood to embrace physical, chemical, photochemical,biological, etc. parameters that can be measured by any kind ofmeasuring apparatus, and physical interaction is understood to mean anykind of physical, chemical, photochemical, biological, etc. interactionwhich is primarily aimed at inducing a change in a physical parameter ofthe sample 14, as well as the process of placing any kind of tool orapparatus in the sample 14.

The measuring system 10 comprises a laser scanning microscope 16 themeasuring space of which (i.e. the spatial region within which it ispossible to measure with the microscope) is not distinguished from themeasuring space 12 of the measuring system 10 for the sake ofsimplicity. It is to be noted that the two measuring spaces 12 generallycoincide because the measuring space of the microscope 16 is comprisedby the measuring space 12 of the measuring system 10, furthermore, thelatter typically does not exceed the measuring space of the microscope16 since any other measurement or interaction is conventionallyperformed within the spatial region that can be scanned by themicroscope 16. It is to be noted furthermore that the measuring space 12is defined with respect to the sample stage of the microscope 16, henceit is conceivable that the focal point of the microscope 16 is shiftedrelative to the sample 14 by displacing the sample stage, also themeasuring space can be enlarged with respect to what can be scanned bysimply deflecting the laser beam by moving the sample stage to variouspositions.

The measuring system 10 further includes some kind of a 3-dimensionalvirtual reality (VR) device 18 which creates a 3-dimensional virtualspace 12′ of the measuring space 12 in a real space region 12″ that isspaced from the measuring space 12. Various VR devices 18 are known fromthe art such as stereoscopic and autostereoscopic displays. The conceptof the stereoscopic displays is to display a right side and a left sideimage for the right and the left eye of a user 11 respectively, whichare perceived as a single 3D image by the human brain. If the right sideand the left side image are not projected in a separated way to theright and left eye of the user 11 respectively, then the two images aretypically projected by two light beams having different physicalproperties (for example blue and red light, orthogonally polarised lightbeams, etc.) In this case the display requires an active or passive userdevice 11 a (typically glasses) for separating the right side and theleft side images. The active user device 11 a can be for examplealternating shutter glasses whereas the passive user device 11 b may befor example polarised 3D glasses.

The autostereoscopic displays do not require any kind of user device 11a, because in this case the display is formed such that the user 11 seesonly the right side image with the right eye and sees only the left sideimage with the left eye. For example this is how the parallax controlleddisplays function.

In case of the above displays it is preferred to use head positiontracking, e.g. by applying the technology disclosed in WO2005/116809.This way the two images destined for the two eyes of the user 11 can beimaged by the 3-dimensional display according to the position of theuser's 11 eyes, whereby the displayed virtual space 12′ does not appeardistorted.

Apart from stereoscopic displays other VR devices 18 can be used aswell, for example, such that create the virtual space 12′ as a realimage, which is spatially distinct from the measuring space 12. Suchdisplays are for example holographic displays (see WO9834411), and 3Dlaser projectors that project a 3D image into the air with laser beams.

In the virtual space 12′ created by the arbitrary VR device 18 themeasuring points of the sample 14 that are measured by the measuringsystem 10 are also displayed by the VR device 18. This can be done byfirst scanning in 3D the sample 14 placed in the measuring space 12 ofthe laser scanning microscope 16. Here, the objective is to allow theuser 11 to view the whole of the sample 14 that can only be scannedrelative slowly, and to allow the user 11 to identify and select regionsof interest therein for performing further measurement or interaction.The VR device 18 displays the virtual image of the scanned sample 14within the virtual space 12′, i.e. the VR device 18 displays a virtualsample 14′ for the user 11.

Scanning the whole of the sample 14 in 3D can be carried out for exampleby scanning the sample 14 in a plurality of planes that are orthogonalor transversal to the optical axis t of the laser scanning microscope16, and displaying the scanned planes by the VR device 18 in the virtualspace 12′.

It is also possible to display the 3D array of the scanned sample 14 inthe virtual space 12′ like a fog by applying some kind of a projection(typically maximum intensity projection). For this kind of real timedisplay typically a lot of graphical calculation is required, that ispreferably implemented in the graphics accelerator or GPU of thecomputer.

In another preferred embodiment the scanned sample 14 is displayed asspatial surface element in the virtual space 12′ after suitablecalculations and one or more simulated light sources are applied toincrease the 3-dimensional effect.

The user 11 may select operations on the virtual sample 14′ displayed inthe virtual space 12′. The measuring system 10 preferably comprises a 3Dinput device 20 for selecting an operation. Such input devices 20 areknown from the art, for example: 3D pointing device in the form of anarm, gesture control recognised by a camera, active/passive markers onthe hand or a handheld device that are detected by an infra red camera(such markers are disclosed in WO2005/116809), etc. The measuring system10 may include one or more 3D input devices 20 (markers) and the appliedmarkers can be of different types, e.g. pen, glove, ball, pin, etc. In apreferred embodiment of the measuring system 10 according to theinvention a pen like 3D marker allowing for absolute positioning withsix degrees of freedom (see WO2005/116809) is used as the input device20. The 3D marker can be displayed in the virtual space 12′ as aconventional cursor. Application of the above mentioned head positiontracking has the advantage of making it possible to display the virtualcursor in the virtual space 12′ so as to coincide exactly with the realspatial position of the 3D marker (i.e. the position perceived by theuser 11 when moving the marker in the real space region 12″).

The measuring system 10 allows for real time bidirectional connectionbetween the measuring space 12 and the virtual space 12′. This meansthat an operation selected in the virtual space 12′ is carried out inthe measuring space 12 and the data measured in the measuring space 12is displayed in the virtual space 12′.

Such an operation can be the selection of a virtual configuration 22′ inthe sample's 14 virtual image within the virtual space 12′ (preferablyby a 3D input device 20), the virtual configuration 22′ consisting ofone or more points and/or curves and/or regions, and commanding thelaser scanning microscope 16 to scan a real configuration 22corresponding to the virtual configuration 22′ in the measuring space12. In order to do this a controlling system 24 (typically computer or acontrolling program running on the computer) of the measuring system 10calculates the coordinates of the real configuration 22 corresponding tothe virtual configuration 22′ in the measuring space 12, and controlsthe laser scanning microscope 16 so as to scan the sample 14 along orwithin the real configuration 22 corresponding to the selected virtualconfiguration 22′. This involves moving the focal point of the laserscanning microscope 16 along or within the real configuration 22, whichcan be performed by deflecting the laser beam and changing the focaldepth and/or by displacing the objective of the microscope 16 and/or bydisplacing the sample stage. Scanning a given point or scanning along agiven curve or within a given region of the sample 14 by the laserscanning microscope 16 can be carried out for example as disclosed inWO2010/007452 or WO2010/055362 or WO2010/055361.

In the course of scanning the real configuration 22 the measurement dataobtained by the microscope 16 can be displayed by the VR device 18 inthe virtual space 12′ in real time, thus the user 11 obtains real timefeed-back from the performance of the selected scanning operation.

The user can also place virtual markings 21′ in the virtual space 12′with the help of the 3D input device 20, in respect of which he mayoptionally select an operation. For example the virtual marking 21′ mayrelate to a certain property or the selection/marking of thecorresponding marked point 21 in the measuring space 12 for any reason.

Preferably, the user 11 is not only able to select operations to beperformed by the laser scanning microscope 16 within the measuringsystem 10. Preferably, the measuring system 10 also comprises one ormore other physical operation devices 26 for performing a physicaloperation in the measuring space 12 such as the performance of aphysical interaction or the measurement of physical parameters. Forexample a suitable operation device 26 may serve to apply mechanical orlaser manipulation in the measuring space 12 of the microscope 12 or itmay serve to control the supply/addition of chemicals or othersubstances locally to the sample 14 or to release substances locallywithin the sample 14. For example the operation selected by the user 11in the virtual space 12′ can be the selection of stimulation points andthe operation device 26 can serve to carry out the stimulation. Thestimulation may be achieved for example by optical stimulation, byprovoking mechanical or electronic response or by injecting a chemicalreagent.

The operation device 26 of the measuring system 10 serving to carry outa physical interaction can be a robot surgical device, such as a knife,a laser ablation device, ultrasound coagulator, laser coagulator, microinjector, vacuum suction device, optical cable, electric stimulator,etc. or a measuring device to be inserted in the sample 14 such as anelectrode, micro pipette used for patch-clamp technique, endoscopicdevice or an electrophoretic measuring head. The operation device 26 maybe suitable for carrying out targeted laser microsurgery at a cell oraxon level.

The operation device 26 may also serve to perform various specificmeasurements. The specific measurement is generally a 3D measuringprocess that is narrowed down and thus accelerated in time, and can becarried out in a smaller spatial region or along a sub-surface or curveor at single points. For example in the case of patch-clamp measurementthe firing pattern of the cells can be traced at higher speeds as well.Such measurements are often combined into protocols that includeapplying a stimulus and analysing the response thereto. The stimulus canbe an electrode provoking an electric stimulus response, or a sensorystimulus, ionthophoresis or local photochemical release of substances,photostimulation of light sensitive proteins or various local shocks.

Preferably, the VR device 18 also displays the operation device 26within the virtual space 12′ in some form, for example a tool 28 of theoperation device 26 performing the physical interaction is representedby a virtual tool 28′ in the virtual space 12′. It is also conceivablethat operation device 26 has no concrete interacting tool 28 (e.g. inthe case of a photostimulation device) but even then it is possible todetermine an interaction point 29 as the point of interaction with thesample 14 and this can be displayed by the VR device 18 in the form of avirtual interaction point 29′.

Preferably, the user 11 may select operations for the operation device26 as well with the help of the 3D input device, for example a virtualconfiguration 22′ may be selected in order to move the interaction point29 of the operation device 26 along or within the real configuration 22corresponding to the selected virtual configuration 22′. Preferably theuser 11 may grab the virtual tool 28′ or virtual interaction point 29′with the help of the cursor of the input device 20, and may drag this toa desired point of the virtual sample 14′ in order to command theoperation device 26 to perform a physical operation at the correspondingpoint of the measuring space 12.

Preferably the controlling system 24 of the measuring system 10 alsocontrols the operation device 26, i.e. the controlling system 24transmits the operation selected in the virtual space 12′ to theoperation device 26 as a command that can be carried out by the latter.

When performing the selected operation by the operation device 26, itmay also provide measurement data (e.g. an electrode can measure thepotential of a cell membrane) which may be displayed real time in thevirtual space 12′ by the VR device 18.

If the measuring system 10 is used to carry out complex operations itmay occur that the measuring system 10 provides data faster than as itcan be interpreted by the user 11. It is also possible that controllingthe operation itself (e.g. measurement/surgery) requires so manydecisions, interactions that it cannot be handled by a single person.For such cases it is preferred to provide a measuring system 10 formultiple users 11. This can be achieved by using a VR device 18 that iscapable of providing virtual reality for a plurality of users 11 or aseparate VR device 18 may be provided for each user 11 in order todisplay the virtual space 12′. The latter embodiment is illustrated inFIG. 2 a. Two separate VR devices 18 create two separate virtual spaces12′ of the measuring space 12 for two users 11, and preferably each user11 can select operations independently using the 3D input device 20provided for the given VR device 18. The controlling system 24preferably translates the operations selected in both virtual space 12′to commands that control the laser scanning microscope 16 and theinteraction tool 28 of any optional operation device 26 in accordancewith the selected operations. It may be advantageous to allocate thecontrol of the microscope 16 and of the interaction tools 28 of theoperation devices 26 between the users 11 in a multi-user measuringsystem 10. For example in the embodiment illustrated in FIG. 2 a thefirst user 11 controls the laser scanning microscope 16 while the seconduser 11 controls the operation device 26. However, it is alsoconceivable that both users 11 control the same measuring instruments(microscope 16 and operation device 26) but in time sharing, making useof the fact that the measurement requires substantially less time thanviewing of the results by the users 11. Time sharing can be applied forexample when the aim is to perform some kind of checking on a largenumber of cells within the volume of the sample 14 in which casemultiple users 11 can perform the checking simultaneously therebysharing the work.

The opposite of work sharing is also conceivable as illustrated in FIG.2 b. If different parts or aspects of the sample 14 are measured withseparate measuring instruments (microscope 16 and other operation device26) the data provided by those can be displayed by a single VR device18, thus allowing the user 11 to decide based on more than onesynchronised information and to control more types of or more complexinteractions and operations. This technique may be applied for examplefor the simultaneous 3D measurement of the cells of the retina and ofthe visual cortex on the opposite side of the head processing the imageof the retina.

The connection between the VR device(s) 18 and the microscope 16 andother optional operation device(s) 26 may be a direct connection throughlocal or global network, the latter including the Internet. For examplea user 11 having special expertise in a certain scientific field cananalyse the sample 14 or control the measurements performed on thesample 14 from a remote location of the world. In the case of remoteconnection it is also possible to allow access to the same measuringsystem 10 for multiple users 11 or to allow a single user 11 to controlmore than one measuring systems 10 (see FIG. 2 c).

The measuring system 10 according to the invention can be used asfollows.

The user 11 places the sample 14 into the measuring space 12 of thelaser scanning microscope 16, and scans the sample 14 in 3D with themicroscope 16. This may be performed for example by scanning the sample14 along a plurality of planes perpendicular to the optical axis t.Following this the VR device 18 displays the scanned sample 14 in thevirtual space 12′ in the form of a virtual sample 14′. The user 11 canpreferably manipulate the virtual sample 14′ in the virtual space 12′with the help of the input device 20 that can be in the form of a 3Dmarker. For example the user 11 may grab the virtual sample 14′,spatially rotate it, displace it, enlarge it or decrease its size,select operations on the virtual sample 14′ to be carried out by themicroscope 16 or optionally by other operation devices 26 belonging tothe measuring system 10. For operations to be carried out by themicroscope 16 the user 11 preferably selects a virtual configuration 22′consisting of one or more points and/or curves and/or regions with thehelp of the 3D marker. The measuring system 10 calculates thecoordinates of the corresponding real configuration 22 in the measuringspace 12, and based on the calculated coordinates the controlling system24 controls the laser scanning microscope 16 so as to scan the sample 14along or within the real configuration 22 corresponding to the selectedvirtual configuration 22′. The measured data can be displayed in thevirtual space 12′ by the VR device 18 of the measuring system 10 in realtime based on which the user 11 can interfere even in the course of themeasurement. Thus, there is real time connection between the measuringspace 12 and the virtual space 12′. The user 11 preferably controls notonly the laser scanning microscope 16 in the virtual space 12′ but alsoother physical operation devices 26, which belong to the measuringsystem 10 as discussed before. The controlling system 24 provides forcarrying out the operation selected in the virtual space 12′ by theoperation device 26 in the measuring space 12, while at the same timethe VR device 18 of the measuring system 10 displays the data measuredin the measuring space 12 by the operation device 26 and/or the laserscanning microscope 16. Hence, there is real time connection between themeasuring space 12 and the virtual space 12′ in this case as well, thusthe user 11 may interfere even in the course of performing the selectedoperation, in real time, and may modify the measurement or selectfurther measurements.

In a preferred embodiment the measuring system 10 operates at one ormore frequencies simultaneously (at the same time) or nearlysimultaneously. In this case the measurement (2-photon fluorescenceintensity, fluorescence life time, transmission signal, second harmonicsgeneration (SHG) signal, polarisation signal, etc.) can be performed atone or more frequencies while the physical operation (photochemicalactivation of molecules, light activated chemical or biologicalsubstances, photo activation of proteins, ablation, etc.) is carried outsimultaneously or nearly simultaneously at one or more differentfrequencies. Due to the application of different frequencies thesimultaneous physical operation and the measurement does not disturbeach other, both can be traced and displayed in the 3D virtual realityenvironment.

Displaying the measurements and other physical operations in 3D virtualreality greatly enhances the effectiveness of controlling the laserscanning microscope 16 and other physical operation devices 26, becausethe user 11 can easily and naturally perceive the abstract or greatlyenlarged information of the virtual space 12′ that is presented to theuser 11 in the form of a virtual reality environment, thus the user 11can perform 3D selections necessary for the 3D operations and carry outthe controlling thereof in a natural way. The most important tasks thatare assisted by the measuring system 10 and method according to theinvention are:

-   -   viewing and understanding of the 3D data structure,    -   orientation of the sample 14 and/or viewpoint and navigation,    -   selection of measuring configurations (planes, curves, volumes)        and setting various parameters therein,    -   controlling special high speed measuring algorithms (see e.g.        WO2010/055361, WO2010/007452) and performing selections within        the measuring space 12 of the sample 14 for this purpose,    -   selection and control of other operations by other operation        devices 26,    -   showing location specific measured data in the virtual space        12′,    -   checking the effect of interactions.

The invention allows for real time selection of the precise spatialposition of a measurement in the measuring space 12 via a suitable 3Dinput device 20 (marker) operating in the corresponding virtual space12′, and allows for the continuous modification thereof. Consequently,the laser scanning microscope 16 need only scan a substantially smallervolume repeatedly, since the measurement is restricted to the realconfiguration 22 corresponding to the manually selected virtualconfiguration 22′. This method allows for performing examinations atmuch higher speeds as compared to scanning the whole sample 14 eachtime. The user 11 may even follow moving objects with the help of themeasuring system 10 (e.g. follow the trajectory of the transportedlabelled particles along the axons of nerve cells).

The precise 3D orientation is vital both in clinical applications and inscientific research when carrying out special measurements or performinginteractions as described above via the operation device 26.

Another advantage of the invention is that displaying the objects in thevirtual space 12′ (e.g. in the case of nerve cells) enhances theinterpretation of formulas similarly to post anatomic reconstructionsbut in real time, thereby helping, for example in the case of nervecells, the recognition of axons, dendrites, dendrite segments or axonsegments that are important from the point of view of the measurement.The virtual reality environment (VR environment) also provides a fastand natural way of selecting structures to be measured, for example itis possible to show the axon to be measured and to set quickly variousmeasuring properties in 3D via the 3D input device 20 of the VR device18. By refreshing only the 3D image points along the selected virtualconfiguration 22′ it is possible to select and set the suitablemeasuring protocol in 3D, i.e. to define where to measure and for howlong. Apart from controlling the measurement of the microscope 16 it isalso possible to control via the measuring system 10 performance ofphotochemical stimulations, mechanical, laser manipulations within themeasuring space 12 by the microscope 16 as well as the local addition(injection) of various chemical compounds and substances. The responses,measured points, curves, regions (i.e. the real configuration 22selected by the user 11) can be displayed in the virtual space 12′ in alocation specific way (e.g. by showing 2D transients on 3D flags linkedto the measured point, or by applying colour coded displaying theactivity map of the measured real configuration 22), whereby thespatiality of the results can be better illustrated.

It should be appreciated that various modifications of the abovedescribed embodiments will be apparent to a person skilled in the artwithout departing from the scope of protection determined by theattached claims.

1. Method for the 3-dimensional measurement of a sample with a measuringsystem having a 3-dimensional measuring space and comprising a laserscanning microscope, characterised by providing the measuring systemwith a 3-dimensional virtual reality device, creating, using the3-dimensional virtual reality device, the 3-dimensional virtual space ofthe measuring space in a real space region that is spaced from themeasuring space, allowing for selecting an operation in the virtualspace, providing real-time bidirectional connection between themeasuring space and the virtual space such that an operation selected inthe virtual space is performed in the measuring space and data measuredin the measuring space is displayed in the virtual space.
 2. The methodaccording to claim 1, characterised by scanning a sample or a portion ofa sample placed in the measuring space of the laser scanning microscopein 3-dimension, displaying the scanned sample in the virtual space,allowing for selecting a virtual configuration comprising at least apoint and/or curve and/or region, calculating the measuring spacecoordinates corresponding to the virtual configuration, based on themeasuring space coordinates controlling the laser scanning microscope soas to scan the sample according to the virtual configuration and toperform measurement and/or interact with the sample in the course ofscanning.
 3. The method according to claim 2, characterised by that thevirtual configuration is a quasi continuous curve consisting of pointsin 3-dimension or a curve interpolated on the defining points andcomprising the steps of: displacing the points of the curve or thedefining points of the interpolated curve in the virtual space; andmodifying the measuring coordinates in the measuring space in accordancewith the displacement.
 4. The method according to claim 2, characterisedby that the virtual space is a quasi continuous curve consisting ofpoints in 3-dimension or a curve interpolated on the defining points andcomprising the steps of: controlling functions and/or algorithmsaffecting the points of the curve or the defining points of theinterpolated curve; displaying these; and modifying the measuringcoordinates in the measuring space accordingly.
 5. The method accordingto claim 2, characterised by scanning the sample in a plurality ofplanes that are orthogonal or transversal to the optical axis of thelaser scanning microscope in the course of the 3-dimensional scanning ofthe sample, and displaying the scanned planes in the virtual space. 6.The method according to claim 5, characterised by displaying the scannedsample or portion thereof in the virtual space as spatial fog.
 7. Themethod according to claim 5, characterised by displaying the scannedsample or portion thereof in the virtual space as spatial surfaceelement and using one or more simulated light sources to increase the3-dimensional effect.
 8. The method according to claim 1, characterisedby providing the measuring system with an operation device serving tocarry out physical operation, which operation device may coincide withthe measuring laser beam, selecting an operation in the virtual spacefor the operation device of the measuring space, performing theoperation selected in the virtual space with the operation device in themeasuring space.
 9. The method according to claim 8, characterised bythat the physical operation is a physical interaction or the measurementof physical parameters, and the operation device serving to carry outthe physical operation is a robot surgical device, knife, laser ablationdevice, ultrasound coagulator, laser coagulator, micro injector, vacuumsuction device, optical cable, prism, grid lens, electric stimulator, ameasuring device to be inserted in the sample, preferably electrode,micro pipette for patch-clamp technique, extracellular recordingelectrode, endoscopic device, electrophoretic device.
 10. The methodaccording to claim 1, characterised by providing the 3-dimensionalvirtual reality device with a 3-dimensional input device—preferably a3-dimensional marker—for selecting operations.
 11. The method accordingto claim 8, characterised by providing the 3-dimensional virtual realitydevice with a 3-dimensional mouse or mechanical arm the coordinatesystem of which is synchronised with the virtual space and beingsuitable for drawing and selecting 3-dimensional spatial curves andpoints for selecting operations.
 12. The method according to claim 11,characterised by that the spatial position of the pointing device of the3-dimensional input device provided for selecting operations coincideswith the spatial position of a virtual marker belonging to the device orit is offset relative to it by a given spatial vector.
 13. The methodaccording to claim 8, characterised by changing the optical contrast orvisibility of the virtual reality in a given local virtual environmentof the 3-dimensional input device and optionally its 3-dimensionalvirtual marker (actuated by the input device), thereby preferablyincreasing the contrast for better viewing an interaction.
 14. Themethod according to claim 1, characterised by providing a 3-dimensionaldisplay device—preferably an autostereoscopic display—within the3-dimensional virtual reality device in order to display data measuredin the measuring space.
 15. The method according to claim 1,characterised by providing the 3-dimensional virtual reality device witha 3-dimensional display device—preferably a stereoscopic display—and anactive or passive stereoscopic user device—preferably glasses—in orderto display data measured in the measuring space.
 16. The methodaccording to claim 14, characterised by providing for head positiontracking within the 3-dimensional virtual reality device and displayingthe virtual space by the 3-dimensional display device in accordance withthe position of the user's head.
 17. The method according to claim 1,characterised by providing the measuring system with more than one3-dimensional virtual reality devices, creating more than one3-dimensional virtual spaces of the measuring space by the more than one3-dimensional virtual reality devices for more than one users.
 18. Themethod according to claim 17, characterised by providing real timeconnection over a local or global network between the more than one3-dimensional virtual reality devices and the measuring system.
 19. Themethod according to claim 1, characterised by providing a common3-dimensional virtual reality device for more than one measuringsystems, and creating a common 3-dimensional virtual space of the morethan one measuring spaces with the common 3-dimensional virtual realitydevice.
 20. The method according to claim 1, characterised by that themeasuring system comprises a two-photon laser scanning microscope or aconfocal laser scanning microscope.
 21. Measuring system for the3-dimensional measurement of a sample, the measuring system having a3-dimensional measuring space and comprising a laser scanningmicroscope, characterised by further comprising a 3-dimensional virtualreality device for displaying a 3-dimensional virtual space of themeasuring space in a real space region that is spaced from the measuringspace, and a real-time bidirectional connection is provided between thelaser scanning microscope and the 3-dimensional virtual reality device.22. The measuring system according to claim 21, characterised bycomprising an operation device for performing physical operation and areal-time unidirectional or bidirectional connection is provided betweenthe operation device and the 3-dimensional virtual reality device. 23.Measuring system for the 3-dimensional measurement of a sample, themeasuring system having a 3-dimensional measuring space and comprising alaser scanning microscope and optionally other operation devices andbeing provided with a controlling system, characterised by comprising a3-dimensional virtual reality device having real-time bidirectionalconnection with the controlling system for performing the methodaccording to claim
 1. 24. The method according to claim 1, characterisedby operating the measuring system at one or more wavelengthssimultaneously or nearly simultaneously.
 25. The method according toclaim 24, characterised by measuring at one or more wavelengths andsimultaneously or nearly simultaneously performing physical operation atone or more different wavelengths.
 26. The method according to claim 20,characterised by providing z-focusing and/or xy-scanning by mechanicaldisplacement in the 3-dimensional laser scanning microscope.
 27. Themethod and measuring system according to claim 1, characterised bydisplaying in the 3-dimensional virtual space the error between thephysical displacement realised in the course of spatial scanning and theconfiguration selected for measuring, and modifying the spatialtrajectory with the 3-dimensional input device so as to decrease thespatial error of the measurement.
 28. The method and measuring systemaccording to claim 20, characterised by using a 3-dimensional laserscanning microscope having an acousto-optic deflector system or having aholographic scanner unit, or by using a 3-dimensional laser scanningmicroscope which performs 3-dimensional scanning by spatio-temporalmultiplexing (i.e. by spatial impulse separation), or by using a3-dimensional laser scanning microscope which performs 3-dimensionalscanning by controlling mechanically displaceable lenses with electricfield.